28 research outputs found
A Mathematical Formalization of Hierarchical Temporal Memory's Spatial Pooler
Hierarchical temporal memory (HTM) is an emerging machine learning algorithm,
with the potential to provide a means to perform predictions on spatiotemporal
data. The algorithm, inspired by the neocortex, currently does not have a
comprehensive mathematical framework. This work brings together all aspects of
the spatial pooler (SP), a critical learning component in HTM, under a single
unifying framework. The primary learning mechanism is explored, where a maximum
likelihood estimator for determining the degree of permanence update is
proposed. The boosting mechanisms are studied and found to be only relevant
during the initial few iterations of the network. Observations are made
relating HTM to well-known algorithms such as competitive learning and
attribute bagging. Methods are provided for using the SP for classification as
well as dimensionality reduction. Empirical evidence verifies that given the
proper parameterizations, the SP may be used for feature learning.Comment: This work was submitted for publication and is currently under
review. For associated code, see https://github.com/tehtechguy/mHT
Analysis of Wide and Deep Echo State Networks for Multiscale Spatiotemporal Time Series Forecasting
Echo state networks are computationally lightweight reservoir models inspired
by the random projections observed in cortical circuitry. As interest in
reservoir computing has grown, networks have become deeper and more intricate.
While these networks are increasingly applied to nontrivial forecasting tasks,
there is a need for comprehensive performance analysis of deep reservoirs. In
this work, we study the influence of partitioning neurons given a budget and
the effect of parallel reservoir pathways across different datasets exhibiting
multi-scale and nonlinear dynamics.Comment: 10 pages, 10 figures, Proceedings of the Neuro-inspired Computational
Elements Workshop (NICE '19), March 26-28, 2019, Albany, NY, US
Convolutional Drift Networks for Video Classification
Analyzing spatio-temporal data like video is a challenging task that requires
processing visual and temporal information effectively. Convolutional Neural
Networks have shown promise as baseline fixed feature extractors through
transfer learning, a technique that helps minimize the training cost on visual
information. Temporal information is often handled using hand-crafted features
or Recurrent Neural Networks, but this can be overly specific or prohibitively
complex. Building a fully trainable system that can efficiently analyze
spatio-temporal data without hand-crafted features or complex training is an
open challenge. We present a new neural network architecture to address this
challenge, the Convolutional Drift Network (CDN). Our CDN architecture combines
the visual feature extraction power of deep Convolutional Neural Networks with
the intrinsically efficient temporal processing provided by Reservoir
Computing. In this introductory paper on the CDN, we provide a very simple
baseline implementation tested on two egocentric (first-person) video activity
datasets.We achieve video-level activity classification results on-par with
state-of-the art methods. Notably, performance on this complex spatio-temporal
task was produced by only training a single feed-forward layer in the CDN.Comment: Published in IEEE Rebooting Computin